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Program Description 1 

University of Chicago School Mathematics Project (UCSMP) is a core 
mathematics curriculum that emphasizes problem solving, real-world 
applications, and the use of technology. The curriculum is based on a 
student-centered approach with a focus on active learning that incor- 
porates reading and uses a flexible lesson organization. This review 
focuses on studies of the following secondary courses: Algebra; 

Geometry; Advanced Algebra; Functions, Statistics, and Trigonom- 
etry; and Precalculus and Discrete Mathematics. 

Research 2 

The What Works Clearinghouse (WWC) identified two studies of 
UCSMP Algebra and one study of the cumulative effect of multiple 
UCSMP courses that both fall within the scope of the Secondary 
Mathematics topic area and meet WWC group design standards. No 
studies meet WWC group design standards without reservations, and 
two studies of UCSMP Algebra meet WWC group design standards 
with reservations. Together, these two studies included 225 students in grades 8-12 in three locations. The one 
study of multiple UCSMP courses also meets WWC group design standards with reservations. This study included 
62 students in grades 7-10 in two locations. 
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This intervention report presents findings 
from a systematic review of USCMP 
conducted using the WWC Procedures 
and Standards Handbook, version 3.0, 
and Secondary Mathematics review 
protocol, version 3.1 . 


The WWC considers the extent of evidence for UCSMP Algebra on the mathematics achievement of secondary 
students to be small for two outcome domains— general mathematics achievement and algebra. The WWC consid- 
ers the extent of evidence for multiple UCSMP courses on the mathematics achievement of secondary students 
to be small for one outcome domain— general mathematics achievement. There were no studies that meet WWC 
group design standards in the four remaining domains, so this intervention report does not report on the effective- 
ness of UCSMP for those domains. (See the Effectiveness Summary on p. 6 for more details of effectiveness by 
domain.) 


The findings in this report pertain to UCSMP Algebra and the cumulative effect of multiple UCSMP courses. No 
studies that independently examine UCSMP Geometry; UCSMP Advanced Algebra; UCSMP Functions, Statistics, 
and Trigonometry; or UCSMP Precalculus and Discrete Mathematics fall within the scope of the Secondary Math- 
ematics review protocol and meet WWC group design standards. 
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Effectiveness 

UCSMP Algebra was found to have potentially positive effects on general mathematics achievement and algebra 
for secondary students. 

The cumulative effect of multiple UCSMP courses was found to have potentially positive effects on general math- 
ematics achievement for secondary students. 

Table 1. Summary of findings 3 


Improvement index 

(percentile points) 


Course and outcome domain 

Rating of 
effectiveness 

Average 

Range 

Number of 
studies 

Number of 
students 

Extent of 
evidence 

UCSMP Algebra 

General mathematics 
achievement 

Potentially positive 
effects 

+29 

na 

1 

189 

Small 

Algebra 

Potentially positive 

+7 

-6 to +20 

2 

225 

Small 


effects 


Multiple UCSMP courses 

General mathematics 
achievement 

Potentially positive 
effects 

+23 

na 

1 

62 

Small 

UCSMP Geometry 

No evidence 






UCSMP Advanced Algebra 

No evidence 






UCSMP Functions, Statistics, 
and Trigonometry 

No evidence 






UCSMP Precalculus and 

Discrete Mathematics 

No evidence 







na = not applicable 
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Program Information 

Background 

UCSMP was developed by the University of Chicago and is published by the University of Chicago School Math- 
ematics Project. Address: 1225 East 60th Street, Chicago, IL 60637. Email: ucsmp@uchicago.edu. Web: http:// 
ucsmp.uchicago.edu. Telephone: (773) 702-1130. Fax: (773) 702-3114. 

Program details 

UCSMP was designed to support student development of deeper conceptual mathematical understanding. To that 
end, the curriculum presents students with challenging mathematics and aids in student acquisition of skills and 
concepts. The curriculum focuses on bringing the real world into the classroom by emphasizing reading, problem 
solving, everyday applications, and the use of calculators, computers, and other technologies. Repetition and 
review are used sparingly. 

Each UCSMP course includes a student textbook, teacher’s edition, teacher resources, assessment resources, and 
technology resources. Lessons in the student book contain activities, full examples, and partially completed guided 
examples to model skills and problem solving. Students are encouraged to assess their own understanding with 
an End-of-Chapter Self-Test correlated to objectives. Projects provided at the end of each chapter are designed as 
extended activities, giving students experience using real data. The use of technology— including graphing calcu- 
lators at all grade levels, geometry systems, spreadsheets, the Internet, and other computer applications— is an 
essential component of the curriculum. 

The publisher is currently selling the third edition of UCSMP. The publisher’s website describes how the curriculum 
has been refined and enhanced from the first and second editions. 


Cost 

The cost of UCSMP varies by course, each of which can be purchased individually. As of December 2015, the 
prices for the student edition of each course ranged from $69.00 to $72.00, the teacher’s edition cost $1 19.00, 
the teacher’s resources cost $89.00, the assessment resources cost $89.00, the electronic teacher’s edition cost 
$1 19.00, the Teacher’s Assessment Assistant cost $89.00, and the ExamView® Assessment Suite cost $89.00. 
These prices are based on quantities of up to 150 for each course; prices could vary for quantities greater than 150. 
More cost information is available from the publisher’s website. 
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Research Summary 

This research summary includes information from studies of all avail- Table 2. Scope of reviewed research 
able UCSMP courses for secondary students. 

The WWC identified 12 studies that were eligible for review: 

• Three eligible studies investigated the effects of UCSMP Algebra 
on the mathematics achievement of secondary students. 

• Four eligible studies investigated the cumulative effect of 4 years of multiple UCSMP courses (including 
UCSMP Transition Mathematics, UCSMP Algebra, UCSMP Geometry, and UCSMP Advanced Algebra) on the 
mathematics achievement of secondary students. 

• Five eligible studies investigated the effects of UCSMP Geometry; UCSMP Advanced Algebra ; UCSMP Functions, 
Statistics, and Trigonometry; and UCSMP Precalculus and Discrete Mathematics on the mathematics achieve- 
ment of secondary students. 4 

The WWC reviewed the 12 eligible studies against group design standards. None of the 12 studies are randomized 
controlled trials that meet WWC group design standards without reservations. Three of the 12 studies are random- 
ized controlled trials or quasi-experimental designs that meet WWC group design standards with reservations. 
Those three studies focus on UCSMP Algebra and multiple UCSMP courses and are summarized in this report. The 
remaining nine studies do not meet WWC group design standards. 

An additional 44 studies were identified but do not meet WWC eligibility criteria for review in this topic area. Cita- 
tions for all 56 studies are in the References section, which begins on p. 8. 

Summary of studies of UCSMP Algebra meeting WWC group design standards without reservations 

No studies of UCSMP Algebra met WWC group design standards without reservations. 

Summary of studies of UCSMP Algebra meeting WWC group design standards with reservations 

Peters (1992) conducted a randomized controlled trial in which the integrity of random assignment was compro- 
mised because some students did not remain in the study group to which they were randomly assigned— students 
were reallocated between the intervention and comparison groups to accommodate scheduling difficulties and 
student requests for other course offerings. The study demonstrated baseline equivalence on the analysis sample 
and therefore, meets WWC group design standards with reservations. The study investigated the effect of UCSMP 
Algebra on the mathematics achievement of 36 “math talented” eighth-grade students (17 UCSMP Algebra and 
19 comparison) from one junior high school in Nebraska during the 1991-92 school year. The district borders two 
large cities, and its students lived in rural and suburban areas. Students in the intervention group used the UCSMP 
Algebra I first-edition textbook, while students in the comparison group used Saxon Algebra I. 

Thompson et al. (2006) used students’ mathematics ability to match six pairs of classrooms within three schools 
to evaluate the effectiveness of the second edition of UCSMP Algebra. Within each pair, random assignment of 
classrooms to the intervention or comparison condition was not always possible; therefore, this study is treated 
as a quasi-experimental design. The WWC review focuses on the three high schools (X, Y, and Z in the study) that 
compared the use of the UCSMP (second edition) to that of non-UCSMP textbooks. 5 These three high schools, 
located in three different regions of the country and serving three different populations of students, contributed 189 
ninth- through twelfth-grade students to the analysis (85% of the students were in ninth grade). 


Grade 

7-12 

Delivery method 

Whole class 

Program type 

Curriculum 
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Summary of studies of multiple UCSMP courses meeting WWC group design standards without reservations 

No studies of multiple UCSMP courses met WWC group design standards without reservations. 

Summary of study of multiple UCSMP courses meeting WWC group design standards with reservations 

Hirschhorn (1993) conducted a longitudinal, 4-year quasi-experimental evaluation of UCSMP (first edition) in 
three high schools in which both traditional and UCSMP curricula were used. Every student in the intervention 
group received 4 years of UCSMP curricula, starting with Transition Mathematics in seventh grade and ending 
with UCSMP Advanced Algebra in tenth grade. No student in the comparison group participated in any UCSMP 
courses. Intervention students were matched to potential comparison students retrospectively using pretest (sixth 
grade) scores, and outcomes were measured at the end of the 4-year period (tenth grade). Two high schools (B 
and C in the study) contributed 62 students to the analysis. A third high school (A in the study) is excluded from the 
WWC review because baseline equivalence between the intervention and comparison groups could not be estab- 
lished. 
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Effectiveness Summary 

The WWC review of UCSMP Algebra and multiple UCSMP courses for the Secondary Mathematics topic area 
includes student outcomes in six domains: general mathematics achievement; algebra; geometry; statistics and 
probability; trigonometry/precalculus; and calculus. The two studies of UCSMP Algebra that meet WWC group 
design standards reported findings in two of the six domains: general mathematics achievement and algebra. The 
one study of multiple UCSMP courses that meets WWC group design standards reported findings in one of the 
six domains: general mathematics achievement. The findings below present the authors’ estimates and WWC- 
calculated estimates of the size and statistical significance of the effects of UCSMP Algebra and multiple UCSMP 
courses on secondary students. For a more detailed description of the rating of effectiveness and extent of evi- 
dence criteria, see the WWC Rating Criteria on p. 24. 

Summary of UCSMP Algebra effectiveness for the general mathematics achievement domain 

Table 3. Rating of effectiveness and extent of evidence of UCSMP Algebra for the general mathematics 


achievement domain 

Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of positive effects with no 
overriding contrary evidence. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the general 
mathematics achievement domain was a statistically significant positive effect. No studies showed a statistically 
significant or substantively important negative effect. 

Extent of evidence 

Criteria met 

Small 

One study that included 189 students in three schools reported evidence of effectiveness in the general 
mathematics achievement domain. 


One study of UCSMP Algebra that meets WWC group design standards with reservations reported findings in the 
general mathematics achievement domain. 

Thompson et al. (2006) found, and the WWC confirmed (after applying a correction for classroom-level clustering), 
one positive and statistically significant difference between the UCSMP Algebra and comparison groups in the 
general mathematics achievement domain. The WWC characterizes this study finding as a statistically significant 
positive effect. 

Thus, for the general mathematics achievement domain, one study of UCSMP Algebra showed a statistically sig- 
nificant positive effect. This results in a rating of potentially positive effects, with a small extent of evidence. 

Summary of UCSMP Algebra effectiveness for the algebra domain 


Table 4. Rating of effectiveness and extent of evidence of UCSMP Algebra for the algebra domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of positive effects with no 
overriding contrary evidence. 

In the two studies that reported findings, the estimated impact of the intervention on outcomes in the algebra 
domain was: one study showing a statistically significant positive effect and one study showing an indeterminate 
effect. 

Extent of evidence 

Criteria met 

Small 

Two studies that included 225 students in four schools reported evidence of effectiveness in the algebra domain. 
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Two studies of UCSMP Algebra that meet WWC group design standards with reservations reported findings in the 
algebra domain. 

Peters (1992) found, and the WWC confirmed (using a difference-in-differences approach), no statistically signifi- 
cant differences between the UCSMP Algebra and comparison groups in the algebra domain. The WWC character- 
izes these study findings as an indeterminate effect. 

Thompson et al. (2006) found a positive and statistically significant difference between the UCSMP Algebra and 
comparison groups on the odd and even forms of the UCSMP Problem-Solving and Understanding Test, analyzed 
separately. The WWC confirmed this finding, pooling the results from both versions of the form into one statistically 
significant positive effect (after applying a correction for clustering). As a result, Thompson et al. (2006) reported, 
and the WWC confirmed (after correction for clustering), a positive and statistically significant difference between 
the UCSMP Algebra and comparison groups in the algebra domain. The WWC characterizes these study findings 
as a statistically significant positive effect. 

Thus, for the algebra domain, one study of UCSMP Algebra showed a statistically significant positive effect and 
one study showed an indeterminate effect. This results in a rating of potentially positive effects, with a small extent 
of evidence. 

Summary of multiple UCSMP courses effectiveness for the general mathematics achievement domain 


Table 5. Rating of effectiveness and extent of evidence of multiple UCSMP courses for the general 
mathematics achievement domain 


Rating of effectiveness 

Criteria met 

Potentially positive effects 

Evidence of positive effects with no 
overriding contrary evidence. 

In the one study that reported findings, the estimated impact of the intervention on outcomes in the general 
mathematics achievement domain was a statistically significant positive effect. No studies showed a statistically 
significant or substantively important negative effect. 

Extent of evidence 

Criteria met 

Small 

One study that included 62 students in one school reported evidence of effectiveness in the general mathematics 
achievement domain. 


One study of multiple UCSMP courses that meets WWC group design standards with reservations reported find- 
ings in the general mathematics achievement domain. 

Hirschhorn (1993) found no statistically significant effects in one site, and a positive and statistically significant 
effect in another, on the Mathematics Level 1 Achievement test. The WWC pooled the results from both sites and 
found no statistically significant differences between multiple UCSMP courses and comparison groups on the test. 
In addition, Hirschhorn (1993) found statistically significant positive effects in each site on the UCSMP-developed 
Applications Test, and the WWC confirmed this finding, pooling the results from both sites. Taking the two out- 
comes together, after adjusting for multiple comparisons, the WWC characterizes this study finding as a statistically 
significant positive effect. 

Thus, for the general mathematics achievement domain, one study of multiple UCSMP courses showed a statisti- 
cally significant positive effect. This results in a rating of potentially positive effects, with a small extent of evidence. 
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Appendix A.1: Research details for Peters (1992) 

Peters, K. G. (1992). Skill performance comparability of two algebra programs on an eighth-grade 
population (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. 
(UMI No. 9314428) 

Table A1. Summary of UCSMP Algebra findings Meets WWC group design standards 

with reservations 


Study findings 
Average improvement index 

Outcome domain Sample size (percentile points) Statistically significant 


Outcome domain 

Study findings 

Average improvement index 

Sample size (percentile points) Statistically significant 

Algebra 

2 classrooms/36 students -6 No 

Setting 

The study took place in one junior high school in Nebraska. The district borders two large cit- 
ies (Lincoln and Omaha) and has a mix of students living in rural and suburban locations. 

Study sample 

The study included two classrooms of the same eighth-grade teacher (for a total of 36 stu- 
dents) from one junior high school during the 1 991-92 school year. All of the students were 
designated as “math talented” based on teacher recommendations and prior academic 
achievement; all students scored at or above the 87th percentile on the California Achieve- 
ment Test total math battery. Of the total sample, 56% were female (53% intervention and 

58% comparison) and 44% were male (47% intervention and 42% comparison). Students 
were randomly assigned to the sole study teacher’s two classrooms, so the teacher used the 
intervention curriculum in one classroom and the comparison curriculum in the other. However, 
the assignment of students was altered after random assignment to accommodate scheduling 
difficulties and student requests for other course offerings. The analytic sample included 17 
students in the UCSMP Algebra group and 19 students in the comparison group. 

Intervention 

group 

Students in the intervention group were taught using UCSMP Algebra during the 1991-92 school 
year. UCSMP Algebra was developed based on National Council of Teachers of Mathematics 
(NCTM) objectives. It emphasizes questioning and problem-solving skills, use of technology, use 
of non-algebraic mathematic topics (e.g., geometry and probability concepts), reading compre- 
hension, and lessons with real-world applications. Lessons are organized into three sections: an 
introduction of a concept, a reading section with an explanation, and real-life problems. 

Comparison 

group 

Students in the comparison group were taught using Saxon Math Algebra 1. Students participated in 
daily lessons from the curriculum for 1 academic year. In each lesson, the teacher introduced a new 
concept incrementally, and students had opportunities to practice new and past concepts. Each les- 
son was structured to allow 30 minutes of teacher instruction followed by 30 minutes of practice. 

Outcomes and 
measurement 

The primary outcome measure was the Orleans-Hanna Algebra Prognosis Test. This measure 
was administered as a pretest in August 1991 and as a posttest in May 1992. For the pretest 
measure, the author reported both a standardized score and a raw score. Only the standard- 
ized score is used in this review because, per the author, the standardized score allows com- 
parability between the pretest and posttest. 
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Support for 
implementation 


The study also examined four study-generated criterion unit tests designed to descriptively 
measure student understanding of algebraic components; however, the author did not provide 
information on the reliability of these four tests. Accordingly, analyses based on these four 
unit tests are not included in this intervention report. In addition, the study examined students’ 
satisfaction with learning. This outcome is not included in this version of the report because it 
is not in a domain focused on in the Secondary Mathematics topic area. For a more detailed 
description of the eligible outcome measure, see Appendix B. 

The teacher who taught both study groups did not have prior experience with the intervention 
or comparison curricula, but had read extensively about both teaching formats. The teacher 
participated in a 1-week summer workshop on UCSMP Algebra, and in two 1-day workshops 
given by local consultants on both of the curricula used in this study. The author also con- 
ducted weekly monitoring to help maintain implementation integrity. 


University of Chicago School Mathematics Project May 201 6 


Page 14 


WWC Intervention Report 


Appendix A.2: Research details for Thompson et al. (2006) 

Thompson, D. R., Senk, S. L., Witonsky, D., Usiskin, Z., & Kaeley, G. (2006). An evaluation of the second 
edition of UCSMP Algebra. Chicago, IL: University of Chicago School Mathematics Project. 


Table A2. Summary of UCSMP Algebra findings Meets WWC group design standards 

with reservations 




Study findings 

Outcome domain 

Sample size 

Average improvement index 
(percentile points) 

Statistically significant 

General mathematics achievement 

12 classrooms/189 students 

+29 

Yes 

Algebra 

12 classrooms/189 students 

+20 

Yes 


Setting The three high schools (grades 9-12) included in the analytic sample were located on the West 
Coast, in the Northeast, and in the South. School X was a large, ethnically diverse high school 
on the West Coast, serving approximately 2,800 students from inner-city and suburban envi- 
ronments; UCSMP Geometry had previously been used at the school. School Y was a subur- 
ban high school in the Northeast, serving 950 students from a middle- to upper-middle-class 
socioeconomic population; no UCSMP curricula were previously used at this school. School 
Z was a suburban high school of approximately 2,800 in a middle- to upper-middle-class 
neighborhood in the South and serves a large Hispanic community; no UCSMP curricula were 
previously used at this school. 

Study sample Thirteen schools were recruited for the study by advertising in UCSMP and NCTM publica- 
tions. To participate, a school needed to have at least two teachers willing to participate, have 
four sections of algebra (in either middle or high school), and promise to keep classes intact 
for a full year. The study utilized a matched-pairs design, in which classes were matched 
within schools on students’ math ability. When possible, classrooms were randomly assigned 
to intervention or comparison conditions within each pair; however, local conditions did not 
always permit random assignment. 

This intervention report focuses on three high schools: X, Y, and Z (as labeled by the authors) 
that compared UCSMP Algebra (second edition) to other curricula. The analysis sample 
size included 189 students (98 intervention and 91 comparison) across six matched pairs of 
classrooms. Twenty-eight students were in one matched pair of classrooms in School X (14 
intervention and 14 comparison), 1 14 in three matched pairs of classrooms in School Y (65 
intervention and 49 comparison), and 47 in two matched pairs of classrooms in School Z (19 
intervention and 28 comparison). 6 About 160 students were in grade 9; the remaining students 
were enrolled in grades 10-12. 


Intervention Intervention classes used UCSMP Algebra (second edition, field trial version) during the 

group 1992-93 school year. UCSMP Algebra emphasizes lessons with real-world applications, use 

of technology (e.g., scientific calculators), spaced introduction of important algebra concepts, 
integration of non-algebraic mathematic topics (e.g., geometry, data organization, and proba- 
bility), and the use of reading passages to explain concepts and provide important information 
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Comparison 

group 


Outcomes and 
measurement 


Support for 
implementation 


The comparison classroom in School X used Saxon Math Algebra I: An Incremental Develop- 
ment. In School Y, classrooms used Houghton Mifflin’s Algebra: Structure and Method Book I, 
and School Z classrooms used Prentice Hall’s Algebra I. 

Shortly before the end of the school year, the teachers administered several assessments, 
three of which are eligible for this review. Two of the eligible assessments are in the algebra 
domain: The High School Subjects Tests: Algebra and a developer-created test— the UCSMP 
Problem-Solving and Understanding Test. The third outcome measure is in the general math- 
ematics achievement domain: the UCSMP Algebra Test. 

In addition to these three eligible outcomes, the study reported results for individual test items 
and for subsets of items based on whether teachers reported having an opportunity to cover 
related content in their classrooms. No measures of reliability or internal consistency were 
provided for these subtest results; consequently, these subtests are not included in this report. 
The study also reported on survey responses about teacher and students’ attitudes. The sur- 
vey data are not in an eligible domain in the Secondary Mathematics topic area and therefore, 
are excluded from this report. 

The pretest used to match students and establish baseline equivalence between the interven- 
tion and comparison groups was the Iowa Algebra Aptitude Test. For a more detailed descrip- 
tion of these outcome measures, see Appendix B. 

Teachers received the UCSMP text in three sections: chapters 1-4 at the beginning of the 
school year, chapters 5-8 in November, and chapters 9-13 in early winter. They also were 
given lesson notes and answers to questions, by chapter, throughout the school year. Teach- 
ers met twice with developers to provide feedback, raises issues, and discuss instructional 
concerns. They did not receive direct in-service training. 
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Appendix A.3: Research details for Hirschhorn (1993) 

Hirschhorn, D. B. (1993). A longitudinal study of students completing four years of UCSMP mathematics. 
Journal for Research in Mathematics Education, 24(2), 136-158. 

Additional source: 

Hirschhorn, D. B. (1991). Implementation of the first four years of the University of Chicago School 
Mathematics Project secondary curriculum (Unpublished doctoral dissertation). University of 
Chicago, IL. 


Table A3. Summary of multiple UCSMP courses findings Meets WWC group design standards 

with reservations 




Study findings 

Outcome domain 

Sample size 

Average improvement index 

(percentile points) Statistically significant 

General mathematics achievement 

2 sites/62 students 

+23 Yes 


Setting The analytic sample included in this intervention report consisted of two unnamed sites in afflu- 
ent suburban areas. Each site included one feeder school in which a subset of the seventh- and 
eighth-grade students received instruction using UCSMP Transition Mathematics and UCSMP 
Algebra and one high school in which a subset of students used UCSMP Geometry and UCSMP 
Advanced Algebra in ninth and tenth grade. 

Study sample Three sites (A, B, and C) were selected where both traditional curricula and UCSMP curricula 
were used by students for 4 years. Students in the intervention group used UCSMP for 4 
years, starting with UCSMP Transition Mathematics in seventh grade and ending with UCSMP 
Advanced Algebra. No student in the comparison group attended any UCSMP courses. 7 The 
researcher matched students on the pretest retrospectively. Since the intervention began in 
seventh grade, students were matched on their sixth-grade performance on a standardized 
exam (in 1986). The process differed slightly by site. In site A, the only available data on sixth- 
grade academic performance were composite scores for mathematics, reading, and general 
logic; because baseline equivalence on mathematics achievement alone could not be estab- 
lished, this site is excluded from WWC analysis. In sites B and C, students were matched on 
sixth-grade mathematics and reading scores. Since some of the comparison students in sites 
B and C had not taken advanced algebra by the end of tenth grade, two comparison groups 
were formed. This report focuses on the age-based comparison sample, wherein the com- 
parison group was based on all students who started seventh grade at the same time (and 
thus, were in tenth grade at the time of the posttest, in 1990). 8 Results from sites B and C were 
pooled for the purposes of this review because when sites were analyzed separately, all stu- 
dents in the intervention group in site B had the same teacher and class in at least one grade, 
creating a confounding factor. 

The eligible sample for this intervention report included 62 students in site A (31 intervention 
and 31 comparison), 26 students in site B (13 intervention and 13 comparison), and 36 stu- 
dents in site C (18 intervention and 18 comparison). 9 Among the sample, 48% were male. 
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Intervention 

group 

Comparison 

group 


Outcomes and 
measurement 


Support for 
implementation 


The intervention students used UCSMP for 4 consecutive years, starting in seventh grade and 
ending in tenth grade. Students used UCSMP Transition Mathematics in 1986-87, UCSMP Algebra 
in 1987-88, UCSMP Geometry in 1988-89, and UCSMP Advanced Algebra in 1989-90. 

Students in the comparison group attended the same high schools and feeder schools as their 
peers in the intervention group. They used traditional mathematics curricula that corresponded 
to each of the UCSMP courses. The traditional curricula were published by Addison-Wesley, 
Merrill, Houghton Mifflin, and unnamed others. 

The study included two outcomes in the general mathematics achievement domain measured 
when students were in the tenth grade, near the end of the academic year (April/May 1990). 
The outcomes were the Mathematics Level I Achievement Test (The College Board, 1988) and 
a developer-created mathematics achievement test. In addition to these two student achieve- 
ment outcomes, the study included one ineligible outcome created using items from the Stu- 
dent Opinion Survey which measured students’ attitudes towards mathematics. 

For the pretest measure, the researchers administered the Stanford Achievement Test, Inter- 
mediate 2 Level - Form E and the Otis Lennon School Ability Test, Intermediate Level, Form 
R at one of the study sites (reading comprehension and total mathematics percentile scores), 
and the McGraw-Hill Comprehensive Test of Basic Skills, Form U, Level G (total reading and 
mathematics percentile scores) at the other study site. For a more detailed description of 
these outcome measures, see Appendix B. 

The report did not describe teacher training or implementation support. However, at site B, 
some of the seventh- and eighth-grade teachers were involved with the UCSMP pilot effort, 
which meant they had previous experience implementing UCSMP curricula. 
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Appendix B: Outcome measures for each domain 


General mathematics 
achievement 

Mathematics Level 1 
Achievement Test, Form 3JAC2 

The Mathematics Level 1 Achievement Test, Form 3JAC2 is a standardized test of mathematics achievement (The 

College Board, 1988) which covers geometry and second year algebra content. The test includes 50 multiple-choice 
questions. The test was scored according to College Board specifications and yielded scores ranging from 200 to 800 
points. No calculators were allowed during the test. The reported reliability alpha statistics ranged from .60 to .93, with a 
median of .87 (as cited in Hirschhorn, 1993). 

UCSMP Algebra and Probability 
Test 

The UCSMP Algebra and Probability test is a 40-item multiple choice test developed by the UCSMP. The topics it covers 
include: translating expressions from verbal to symbolic, linear relationships in two variables, quadratics, geometric rela- 
tionships, statistics and probability, percentage applications, graph interpretation, exponential relationships, and other 
miscellaneous topics. Students were allowed to use calculators during the test. Although this assessment was designed 
by the developer of the curriculum used in the intervention condition, it focuses on content that is typically taught in 
algebra classrooms, so these assessments are not considered overaligned with the intervention. Because the content of 
the assessment includes algebra, geometry, and statistics and probability, the test falls within the general mathematics 
achievement domain per the Secondary Mathematics review protocol (version 3.1). The reported Kuder-Richardson 
reliability estimate ranged between .81 and .83 (as cited in Thompson et al., 2006). 

UCSMP Applications Test 

The UCSMP Applications Test is a 30-question, multiple-choice test that was developed for this study, as well as for use 
by future researchers and school personnel. It covered the topics of arithmetic, algebra, geometry, and advanced alge- 
bra. Scientific calculators were provided to students who did not have them, and their use was encouraged. Items were 
selected from the Second International Mathematics Study (Chang & Ruzicka, 1985), the Formative Study of UCSMP 
Advanced Algebra (Hedges et al., 1988), a submittal to the College Board by personal letter in 1988, and original items. 
Reported sample reliability alpha statistics ranged from .64 to .84, with a median of .74 (as cited in Hirschhorn, 1993). 

Algebra 

Orleans- Hanna Algebra 

Prognosis Test 

The 60-item nationally normed Orleans-Hanna Algebra Prognosis Test is used to place and group students in algebra 
courses and inform the development of lessons plans. It was developed in 1928 and revised in 1980. This measure was 
administered as a pretest in August 1991 and as a posttest in May 1992. For the pretest measure, the author reports 
both a standardized score and a raw score. Only the standardized score is used in this review because, per the author, 
the standardized score allows comparability between the pretest and posttest. The reported sample Kuder-Richardson 
reliability estimate was .96 (as cited in Peters, 1992). 

High School Subject Tests: 
Algebra 

The High School Subject Tests: Algebra is a 40-item multiple-choice standardized test developed by American Testronics. 

It includes items on polynomials, linear equations/inequalities, evaluating expressions, linear relationships in two variables, 
quadratics, radicals, properties of numbers, linear systems, rational expressions, factoring, literal equations, and propor- 
tions. The use of calculators was not permitted during the test. The test developer-reported Kuder-Richardson reliability 
estimate was .86, and for the sample it was reported as being approximately .80 (as cited in Thompson et al., 2006). 

UCSMP Problem-Solving and 
Understanding Test 

The UCSMP Problem-Solving and Understanding Test is an open-ended problem-solving test developed by UCSMP. In 
the study, it was administered in two forms, with half the students in each class randomly assigned to the even form of 
the test and the other half assigned to the odd form. Each form contains four open-ended items. The results from the 
two forms were pooled by the WWC. Five of the eight items were scored on a scale of zero to four, and three items on 
a scale of zero to two. Each item was scored independently and blindly by two raters guided by rubrics developed for 
each item, and when they disagreed, a third rater scored the items, and the median score was used. Inter-rater reliability 
estimates (not identified) ranged from .74 to .95 for items on both forms. Although this assessment was designed by 
the developer of the curriculum used in the intervention condition, it focuses on content that is typically taught in algebra 
classrooms, so these assessments are not overaligned with the intervention (as cited in Thompson et al,, 2006). 
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Appendix C.1: Findings included in the rating for studies of UCSMP Algebra lor the general mathematics 
achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Thompson et al. (2006) a 

UCSMP Algebra Test 

Grades 

9-12, 

12 classes/ 
189 students 

49.8 

(16.3) 

37.3 

(14.9) 

12.54 

0.81 

+29 

.04 


Schools X, 

Y, &Z 

Domain average for general mathematics achievement (Thompson et al., 2006) 0.81 +29 Statistically 

significant 


Domain average for general mathematics achievement across all studies 0.81 +29 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study's domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Thompson et al. (2006), the WWC did not need to make corrections for clustering or multiple comparisons. The p-value presented here was reported in the original study. The 
WWC calculated the intervention group mean using a difference-in-differences approach by adding the impact of the program (i.e., difference in mean gains between the interven- 
tion and comparison groups) to the unadjusted comparison group posttest means. Please see the WWC Procedures and Standards Handbook (version 3.0) for more information. The 
findings reported for the UCSMP Algebra Test in the earlier version of this report were adjusted for multiple comparisons because all three of the outcomes from the Thompson et al. 
(2006) study were included in the only outcome domain in the topic area: general mathematics achievement. This report includes multiple outcome domains. As a result, there is only 
one outcome in the general mathematics achievement outcome domain (the other outcomes are in the algebra outcome domain). Because there is only one outcome in the general 
mathematics achievement domain, a multiple comparison correction was not required. The WWC calculations reported in this table differ slightly from those reported in the earlier 
version of this report. The slight difference results from analyses using a difference-in-differences approach, which was not used in the prior version. This study is characterized as 
having a statistically significant positive effect because the estimated effect is positive and statistically significant. For more information, please refer to the WWC Procedures and 
Standards Handbook (version 3.0), p. 26. 
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Appendix C.2: Findings included in the rating for studies of UCSMP Algebra lor the algebra domain 





Mean 

(standard deviation) 

WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-vaiue 

Thompson et al. (2006) a 

High School Subject Tests: 
Algebra 

Mostly grade 

9, Schools X, 

Y, &Z 

12 classes/ 
189 students 

48.24 

(16.30) 

46.00 

(14.90) 

2.24 

0.15 

+6 

.83 

UCSMP Problem-Solving and 
Understanding Test 

Mostly grade 

9, Schools X, 

Y, &Z 

12 classes/ 
189 students 

6.57 

(3.69) 

3.39 

(2.54) 

3.18 

0.92 

+32 

< .02 

Domain average for algebra (Thompson et al., 2006) 




0.53 

+20 

Statistically 

significant 

Peters (1992) b 

Orleans -Hanna Algebra 
Prognosis Test 

Grade 8 
(math 
talented) 

2 classes/ 

36 students 

95.02 

(4.09) 

95.63 

(4.53) 

-0.61 

-0.14 

-6 

> .05 

Domain average for algebra (Peters, 1992) 





-0.14 

-6 

Not 

statistically 

significant 

Domain average for algebra across all studies 




0.20 

+7 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who 
are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change 
in an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to 
two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of each study's domain average was determined by the 
WWC. Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Thompson et al. (2006), corrections for clustering and multiple comparisons were needed but did not affect whether any of the contrasts were found to be statistically significant. 
The WWC calculated the intervention group mean using a difference-in-differences approach by adding the impact of the program (i.e., difference in mean gains between the inter- 
vention and comparison groups) to the unadjusted comparison group posttest means. Please see the WWC Procedures and Standards Handbook (version 3.0) for more information. 
The study-reported results for the UCSMP Problem-Solving and Understanding Test odd and even forms separately. The two p-values associated with the forms are < .01 and .02, 
respectively. The WWC pooled the results across the two forms. The WWC calculations reported in this table differ slightly from those reported in the earlier version of this report. 

The slight difference results from analyses using a difference-in-differences approach, which was not used in the earlier version. This study is characterized as having a statistically 
significant positive effect because the effect for at least one measure within the domain is positive and statistically significant, and no effects are negative and statistically significant 
(correcting for clustering). For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 

b For Peters (1 992), the WWC did not need to make corrections for clustering, multiple comparisons, or to adjust for baseline differences. The p-value presented here was reported in 
the original study. The WWC calculated the intervention group mean using a difference-in-differences approach by adding the impact of the program (i.e., difference in mean gains 
between the intervention and comparison groups) to the unadjusted comparison group posttest means. Please see the WWC Procedures and Standards Handbook (version 3.0) for 
more information. This study is characterized as having an indeterminate effect because the estimated effect is neither statistically significant nor substantively important. For more 
information, please refer to the WWC Standards and Procedures Handbook (version 3.0), p. 26. 
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Appendix C.3: Findings included in the rating for multiple UCSMP courses for the general mathematics 
achievement domain 


Mean 

(standard deviation) WWC calculations 


Outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index p-value 

Hirschhorn (1993) a 

Mathematics Level 1 

Grade 10, 

2 sites/ 

492.36 

489.03 

3.32 

0.06 

+2 

Site B: < .05 

Achievement Test 

sites B & C 

62 students 

(46.39) 

(78.71) 




Site C: > .05 

UCSMP Applications Test 

Grade 10, 

2 sites/ 

21.19 

16.46 

4.74 

1.18 

+38 

Site B: < .01 


sites B & C 

62 students 

(3.88) 

(3.95) 




Site C: < .05 

Domain average for general mathematics achievement (Hirschhorn, 1993) 


0.62 

+23 

Statistically 









significant 

Domain average for general mathematics achievement across all studies 


0.62 

+23 

na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in 
an average individual’s percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average rounded to two 
decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was determined by the WWC. 
Some statistics may not sum as expected due to rounding, na = not applicable. 

a For Hirschhorn (1 993), the author presented p-values as well as unadjusted posttest means and standard deviations by site. The results for the Mathematics Level 1 Achievement 
Test were positive and statistically significant in site B (< .05) and not statistically significant or substantively important in site C (> .05). The results for the UCSMP Applications 
Test were positive and statistically significant at each site (< .01 in site B and < .05 in site C). The WWC combined the results for sites B and C and calculated the intervention 
group mean using a difference-in-differences approach by adding the impact of the program (i.e., difference in mean gains between the intervention and comparison groups) to the 
unadjusted comparison group posttest means. After adjusting for multiple comparisons, the WWC-calculated results were neither statistically significant nor substantively important 
for the Mathematics Level 1 Achievement Test (.81 ) and positive and statistically significant for the UCSMP Applications Test (< .01 ). Please see the WWC Procedures and Standards 
Handbook (version 3.0) for more information. The study was reviewed as evidence of multiple UCSMP courses, following the procedure set in the Secondary Mathematics review 
protocol (version 3.1 ). This approach differs from the approach used when this study was previously reviewed under the High School Mathematics review protocol (version 2.0). The 
WWC calculations reported in this table differ slightly from those reported in the earlier version of this report. The difference results from analyses using a difference-in-differences 
approach, which was not used in the earlier version. This study is characterized as having a statistically significant positive effect because the effect for at least one measure within 
the domain is positive and statistically significant, and no effects are negative and statistically significant, accounting for multiple comparisons. For more information, please refer to 
the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Endnotes 

' Due to the 2015 restructuring of the Mathematics topic area from three areas (Elementary, Middle, and High School) to two areas (Pri- 
mary and Secondary Mathematics), this report is considered a new report rather than an updated report. The information in this report 
combines the research examined in the prior reports and presents the conclusions differently. 

1 The descriptive information for this program was obtained from publicly available sources, such as the publisher website (http:// 
ucsmp.uchicago.edu/secondary/overview/). The WWC requests developers review the program description sections for accuracy 
from their perspective. The program description was provided to the developer in November 2014; however, the WWC received no 
response. Further verification of the accuracy of the descriptive information for this program is beyond the scope of this review. 

2 The WWC previously released two reports for UCSMP: University of Chicago School Mathematics Project Algebra was reviewed 
under the Middle School Mathematics review protocol and released in March 2009, and University of Chicago School Mathematics 
Project 6-12 Curriculum was reviewed under the High School Mathematics review protocol and released in July 201 1 . The literature 
search reflects documents publicly available by November 2015. This report has been updated to include reviews of 23 studies that 
were not included in the prior reports. Of the additional studies, 23 were not within the scope of the review protocol for the Second- 
ary Mathematics topic area. A complete list and disposition of all studies reviewed are provided in the references. The studies in 
this report were reviewed using the Standards from the WWC Procedures and Standards Handbook (version 3.0), along with those 
described in the Secondary Mathematics topic area review protocol (version 3.1). The evidence presented in this report is based on 
available research. Findings and conclusions may change as new research becomes available. 

3 For criteria used in the determination of the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 24. 
These improvement index numbers show the average and range of individual-level improvement indices for all findings across the 
studies. 

4 No studies examining the effectiveness of UCSMP Geometry; UCSMP Advanced Algebra; UCSMP Functions, Statistics, and Trigo- 
nometry; and UCSMP Precalculus and Discrete Mathematics fall within the scope of the Secondary Mathematics review protocol 
and meet WWC group design standards. Because no studies meet WWC group design standards at this time, the WWC is unable to 
draw any conclusions based on research about the effectiveness or ineffectiveness of UCSMP Geometry; UCSMP Advanced Algebra; 
UCSMP Functions, Statistics, and Trigonometry; and UCSMP Precalculus and Discrete Mathematics on secondary students. Addi- 
tional research that meets WWC standards is needed to determine the effectiveness or ineffectiveness of these courses. 

5 The study authors also examined a second comparison group that used UCSMP Algebra first edition; the comparison included 334 
students. The comparison was not eligible for review based on the Secondary Mathematics topic area review protocol and is excluded 
from this intervention report. 

6 The study authors also examined a second comparison group that used UCSMP Algebra first edition; the comparison included 334 
students. The comparison was not eligible for review based on the Secondary Mathematics topic area review protocol and is excluded 
from this intervention report. 

7 UCSMP Transition Mathematics falls within the Primary Mathematics topic area. Because students also used three Secondary Math- 
ematics courses (UCSMP Algebra, UCSMP Geometry, and UCSMP Advanced Algebra) and the student outcomes were measured 
after UCSMP Advanced Algebra, this study is being reviewed by the Secondary Mathematics topic area. 

8 The second course-based comparison group consisted of students at the end of UCSMP Advanced Algebra. Since this involved 
comparing tenth-grade intervention students with eleventh-grade comparison students, we did not include this comparison in the 
intervention report. 

9 The WWC excluded the 22 students in site A (1 1 intervention and 1 1 comparison). In site A, the only available data on sixth-grade 
academic performance were composite scores for math, reading, and general logic; because baseline equivalence on math achieve- 
ment alone could not be established, this site is excluded from WWC analysis. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2016, May). 

Secondary Mathematics intervention report: University of Chicago School Mathematics Project. Retrieved 
from http://whatworks.ed.gov 
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WWC Rating Criteria 

Criteria used to determine the rating of a study 

Study rating 

Criteria 

Meets WWC group design 
standards without reservations 

A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 

Meets WWC group design 

A study that provides weaker evidence for an intervention's effectiveness, such as a QED or an RCT with high 

standards with reservations 

attrition that has established equivalence of the analytic samples. 

Criteria used to determine the rating of effectiveness for an intervention 

Rating of effectiveness 

Criteria 

Positive effects 

Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important negative effects. 

Potentially positive effects 

At least one study shows a statistically significant or substantively important positive effect, AND 

No studies show a statistically significant or substantively important negative effect AND fewer or the same number 
of studies show indeterminate effects than show statistically significant or substantively important positive effects. 

Mixed effects 

At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 

At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 

Potentially negative effects 

One study shows a statistically significant or substantively important negative effect and no studies show 
a statistically significant or substantively important positive effect, OR 

Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 

Negative effects 

Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 

No studies show statistically significant or substantively important positive effects. 

No discernible effects 

None of the studies shows a statistically significant or substantively important effect, either positive or negative. 

Criteria used to determine the extent of evidence for an intervention 

Extent of evidence 

Criteria 

Medium to large 

The domain includes more than one study, AND 

The domain includes more than one school, AND 

The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 

Small 

The domain includes only one study, OR 

The domain includes only one school, OR 

The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in a class, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 

Attrition 

Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Extent of evidence 

Improvement index 

Intervention 
Intervention report 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Rating of effectiveness 


Single-case design 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

An indication of how much evidence supports the findings. The criteria for the extent 
of evidence levels are given in the WWC Rating Criteria on p. 24. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

An educational program, product, practice, or policy aimed at improving student outcomes. 

A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

The WWC rates the effects of an intervention in each domain based on the quality of the 
research design and the magnitude, statistical significance, and consistency in findings. The 
criteria for the ratings of effectiveness are given in the WWC Rating Criteria on p. 24. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 
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Glossary of Terms 


Standard deviation The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 

Statistical significance Statistical significance is the probability that the difference between groups is a result of 

chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 


Substantively important a substantively important finding is one that has an effect size of 0.25 or greater, regardless 

of statistical significance. 

Systematic review a review of existing literature on a topic that is identified and reviewed using explicit meth- 
ods. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching 
the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the 
methodological quality of each study, and reporting on high quality studies and their find- 
ings; 4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Mathematica Policy Research under contract ED-IES-13-C-0010. 
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